The integrated radio and optical communications (iROC) project at the NASA Glenn Research Center (GRC) is investigating the feasibility of a hybrid RF and optical communication subsystem for future deep space missions. The hybrid communications subsystem enables the advancement of optical communications while simultaneously mitigating the risk of infusion by combining an experimental optical transmitter and telescope with a reliable Ka-band RF transmitter and antenna. The iROC communications subsystem seeks to maximize the total data return over the course of a potential 2-year mission in Mars orbit beginning in 2021. Although optical communication by itself offers potential for greater data return over RF, the reliable Ka-band link is also being designed for high data return capability in this hybrid system. A daily analysis of the RF link budget over the 2-year span is performed to optimize and provide detailed estimates of the RF data return. In particular, the bandwidth dependence of these data return estimates is analyzed for candidate waveforms. In this effort, a data return modeling tool was created to analyze candidate RF modulation and coding schemes with respect to their spectral efficiency, amplifier output power back-off, required digital to analog conversion (DAC) sampling rates, and support by ground receivers. A set of RF waveforms is recommended for use on the iROC platform.